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METHOD AND SYSTEM FOR COMPOSITING IMAGES WITH 
COMPENSATION FOR LIGHT FALLOFF 

FIELD OF THE INVENTION 

5 The invention relates generally to the field of digital image 

processing, and in particular to a technique for compositing multiple images into a 
panoramic image comprising a large field of view of a scene. 

BACKGROUND OF THE INVENTION 

1 0 Conventional methods of generating panoramic images comprising 

a wide field of view of a scene from a plurality of images generally include the 
following steps: (1) an image capture step, where the plurality of images of a 
scene are captured with overlapping pixel regions; (2) an image warping step, 
where the captured images are geometrically warped onto a cylinder, sphere, or 

1 5 any environment map; (3) an image registration step, where the warped images 
are aligned; and (4) a blending step, where the aligned warped images are 
blended togeliier to form the panoramic image. For an example of an imaging 
system that generates panoramic images, see May et al. USSN 09/224,547 filed 
December 31, 1998. 

20 In the image capture step, the captured images typically suffer from 

light falloff As described in many texts on the subject of optics (for example, M. 
Klein, Optics, John Wiley & Sons, Inc., New York, 1986, pp. 193-256), lenses 
produce non-uniform exposure at the focal plane when imaging a uniformly 
illuminated surface. When the lens is modeled as a thin lens, the ratio of the 

25 intensity of the light of the image at a point is described as cos"^ of the angle 
between the optical axis, the lens, and the point in the image plane. This cos 
falloff does not include such factors as vignetting, which is a property describing 
the loss of light rays passing through an optical system. 

In photographic images, this cos'* falloff generally causes the 

30 comers of an image to be darker than desired. The effect of the falloff is more 
severe for cameras or capture devices with a short focal length lens. In addition, 
flash photography will often produce an effect similar to falloff if the subject is 
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centrally located with respect to the image. This effect is referred to as flash 
falloff. 

As described in US Patent 5,461,440 issued October 24, 1995 to 
Toyoda et al., it is commonly known that light falloff may be corrected by 
5 applying an additive mask to an image in a log domain or a multiplicative mask to 
an image in the linear domain. This conventional cos based mask is solely 
dependent upon a single parameter: the focal length of the imaging system. Also, 
images with flash falloff in addition to lens falloff, may be compensated for by a 
stronger mask (i.e. a mask generated by using a smaller value for the focal length 

1 0 than one would normally use). 

Gallagher et al. in USSN 09/293,197 filed April 16, 1999 describe 
a variety of methods of selecting the parameter used to generate the falloff 
compensation mask. For example, in this conventional teaching the parameter 
could be selected in order to simulate the level of falloff compensation that is 

1 5 naturally performed by the lens of the optical printer. Additionally, the parameter 
could be determined interactively by an operator using a graphical user interface 
(GUI), or the parameter could be dependent upon the film format (APS or SUC) 
or the sensor size. Finally, they teach a simple automatic method of determining 
the parameter. 

20 Gallagher in USSN 09/626,882 filed July 27, 2000 describes a 

method of automatically determining a level of hght falloff in an image. This 
method does not misinterpret image discontinuities as being caused by light 
falloff, as frequently happens in the other methods. 

In panoramic imaging systems, any of the aforementioned methods 

25 of light falloff compensation could be used to compensate for the light falloff 

present in each source image. However, there would be a problem with using any 
of these methods directly. Since all of the current light falloff compensation 
methods are applicable to single images, any errors in the falloff compensation for 
each source image could be magnified when the composite image is formed. 



Therefore, there exists a need in the art for a method of 
compensating for light falloff in multiple images that are intended to be combined 
into a composite image. 

SUMMARY OF THE INVENTION 
The need is met according to the present invention by providing a 
method and system for producing a composite digital image that includes 
providing a plurality of partially overlapping source digital images having pixel 
values that are linearly or logarithmically related to scene intensity; modifying the 
source digital images by applying to one or more of the source digital images a 
radial exposure transform to compensate for exposure falloff as a function of the 
distance of a pixel from the center of the digital image to produce adjusted source 
digital images; and combining the adjusted source digital images to form a 
composite digital image. 

ADVANTAGES 

The present invention has the advantage of simply and efficiently 
matching source digital images having light falloff characteristics such that the 
light falloff is compensated prior to the compositing step. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram illustrating a digital image processing 
system suitable for practicing the present invention; 

Fig 2 illustrates in block diagram form, tlie method of forming a 
composite image from at least two source images, at least one source image being 
compensated for light falloff; 

Fig. 3 illustrates in block diagram form, one embodiment of the 
present invention; 

Figs. 4A and 4B illustrate the overlap regions between source 

images; 



Figs. 5 A and 5B illustrate in block diagram form, the step of 
providing source digital images to the present invention; 

Fig. 6 is a diagram of the relationship between the focal length, 
pixel position, and light falloff parameter in one of the source digital images; 

Fig. 7 is a diagram of the relationship between the focal length, 
pixel position, and light falloff parameter in two of the source digital images; 

Fig. 8 is a diagram of the process of modifying the source digital 
image to compensate for light fallolf; 

Fig. 9 is a plot of the pixel values in the overlap region of the 
second source digital versus the pixel values of the overlap region of the first 
source digital image; 

Fig. 10 is a plot of the pixel values in the overlap region of the 
second source digital image versus the pixel values of the overlap region of the 
first source digital image; 

Fig . 1 1 is a diagram of the process of combining images to form a 
composite image; 

Fig. 12 illustrates in block diagram form, an embodiment of the 
present invention further including the step of transforming the composite image 
into an output device compatible color space; and 

Figs. 13A and 13B are diagrams of image data and metadata 
contained in a source image file. 

DETAILED DESCRIPTION OF THE INVENTION 
The present invention will be described as implemented in a 
programmed digital computer. It will be understood that a person of ordinary skill 
in the art of digital image processing and software programming will be able to 

program a computer to practice the invention from the description given below. 
The present invention may be embodied in a computer program product having a 
computer readable storage medium such as a magnetic or optical storage medium 
bearing machine readable computer code. Alternatively, it will be understood that 
the present invention may be implemented in hardware or firmware. 
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Referring first to Fig. 1, a digital image processing system useful 
for practicing the present invention is shown. The system generally designated 
10, includes a digital image processing computer 12 connected to a network 14. 
The digital image processing computer 12 can be, for example, a Sun 

5 Sparcstation, and the network 14 can be, for example, a local area network with 
sufficient capacity to handle large digital images. The system includes an image 
capture device 15, such as a high resolution digital camera, or a conventional film 
camera and a film digitizer, for supplying digital images to network 14. A digital 
image store 16, such as a magnetic or optical multi-disk memory, connected to 

10 network 14 is provided for storing the digital images to be processed by computer 
12 according to the present invention. The system 10 also includes one or more 
display devices, such as a high resolution color monitor 18, or hard copy output 
printer 20 such as a thermal or inkjet printer. An operator input, such as a 
keyboard and track ball 21, may be provided on the system. 

1 5 Referring next to Fig. 2, at least two overlapping source digital 

images are provided 200 to the processing system 10. The source digital images 
can be provided by a variety of means; for example, they can be captured from a 
digital camera, extracted from frames of a video sequence, scanned from 
hardcopy output, or generated by any other means. The pixel values of at least 

20 one of the source digital images are modified 202 by a radial exposure transform 
so that any light falloff present in the source digital images is compensated, 
yielding a set of adjusted source digital images. A radial exposure transform 
refers to a fransformation that is applied to the pixel values of a source digital 
image, the transformation being a function of the distance from the pixel to the 

25 center of the image. The adjusted source digital images are then combined 204 by 
a feathering scheme, weighted averages, or some other blending technique known 
in the art, to form a composite digital image 206. 

Referring next to Fig. 3, according to an alternative embodiment of 
the present invention, at least two overlapping source digital images are provided 

30 300 to the processmg system 10. The pixel values of at least one of the source 
digital images are modified 302 by a radial exposure transform so that any light 



falloff present in the source digital images is compensated. In addition, the pixel 
values of at least one of the source digital images are modified 304 by a linear 
exposure transform so that the pixel values in the overlap regions of overlapping 
source digital images are similar. A linear exposure transform refers to a 
transformation that is applied to the pixel values of a source digital image, the 
transformation being linear with respect to the scene intensity values at each 
pixel. The radial exposmre transform and the linear exposure transform can be 
applied to the same source digital image, or to different source digital images. 
Also, the modification steps 302 and 304 can be applied in any order. Once either 
or both of the modification steps are completed, they yield adjusted source digital 
images. The adjusted source digital images are then combined 306 by a 
feathering scheme, weighted averages, or some other blending technique known 
in the art, to form a composite digital image 308. 

Referring next to Figs. 4A and 4B, the at least two source digital 
images 400 overlap in overlapping pixel regions 402. 

Referring next to Fig. 5A, according to a furtitier embodiment of 
the present invention, the step 200 of providing at least two source digital images 
further comprises the step 504 of applying a metric transform 502 to a source 
digital image 500 to yield a transformed source digital image 506. A metric 
transform refers to a transformation that is applied to the pixel values of a source 
digital image, the transformation yielding transformed pixel values that are 
linearly or logarithmically related to scene intensity values. In instances where 
metric transforms are independent of the particular content of the scene, they are 
referred to as scene independent transforms. 

Referring next to Fig. 5B, in one embodiment, the step of applying 
the metric transform 504 includes applying a matrix transformation 508 and a 
gamma compensation lookup table 510. In one example of such an embodiment, 
a source digital image 500 was provided from a digital camera, and contains pixel 
values in the sRGB color space. A metric transform 502 is used to convert the 
pixel values into nonlinearly encoded Extended Reference Input Medium Metric 
(ERIMM) (PIMA standard #7466, found on the World Wide Web at 
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http://www.pima.net/standards/itl O/rr 1 0 PQW.htm) , so that the pixel values are 
logarithmically related to scene intensity values. 

Referring next to Fig. 6, we illustrate the relationship between the 
focal length/ 600, pixel position (u,v) 602, and light falloff parameter 0 604 in 
5 one of the source digital images 608. If tiie origin 606 is located at the center of 
the source digital image 608, and if a uniformly illuminated surface parallel to the 
image plane is imaged through a thin lens, then the exposure i(w, v) received at 
pixel position (m,v) is given by: 
M l(ia, v) = l(0,0)cos^ (tan-^ e), 

o 

rf where l(o,o) is the exposure falling on the center of the source digital image 608, 

U and the focal length/ 600 is measui-ed in tenns of pixels. 

= Referring next to Fig. 7, we illustrate two source digital images 

~! 700 and 702 that overlap in an overlapping pixel region 704. The center 706 of 

= y 

i== 1 5 source digital image 700 is located at the image center, and its local coordinate 

p system is defined by positions (m,v) . The center 708 of source digital image 702 

^~ is located at the image center, and its local coordinate system is defined by 

positions (x,y). The focal length used in the capture of source digital images 700 
and 702 is given by / 710 (in pixels). Consider a point 714 located in the 
20 overlapping pixel region 704. If the coordinates of point 714 are given by (m,-,v, ) 
in image 700, and by (x;,y;) in image 702, and if a uniformly illuminated surface 
parallel to the image plane is imaged through a thin lens during both the captures 
of source digital image 700 and source digital image 702, then the exposure 
l(u;,v,) received at point 714 in source digital image 700 is given by: 
25 l(u{,Vi)=Ii(0,0)cos'^(tan~^ai), 
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where li(0,0) is the exposure falling on the center of the source digital image 700. 
The exposure l(;c, ,>', ) received at point 714 in source digital image 702 is given 
by: 

l(xi,yi) = l2(0>0)cos'^(tan-l Pi), 



5 




where 12(0,0) is the exposure falling on the center of the source digital image 702. 

The point 714 in the overlapping pixel region 704 when considered 
as a point in the source digital image 700, corresponds to the same scene content 
as if it were considered a point in the source digital image 702. Therefore, if the 

1 0 overall exposure level of each source digital image 700 and 702 is the same, then 
the light f alloff can automatically be determined without the knowledge of the 
focal length. (Note that if the focal length is known, the amount of light falloff is 
readily determined by the aforementioned formula). Consider that the exposure 
value recorded at point 714 in source digital image 700 is /,' , and the exposure 

15 value recorded at point 714 in source digital image 702 is . Then, the 
following relation must hold: 




Since // , //' , , v,- , x, , and are known, the focal length / 710 can be found 
by identifying the root of the fiinction: 




This root can be approximated by an iterative process, such as 
Newton's method; see J. Stewart, "Calculus", 2""^ Ed., Brooks/Cole Publishing 
Company, 1991, p. 170. Once the focal length / 710 has been found, we know 
enough information to compensate for light falloff without having to identify the 
25 falloff parameter as described in one of the aforementioned light falloff 
compensation techniques. 
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Even though the focal length can be estimated from the pixel 
values of a single point 714 in the overlapping pixel region 704 of the source 
digital images 700 and 702, multiple points in the overlapping pixel region can be 
used to provide a more robust estimate. Consider n points in the overlapping 

5 pixel region 704, where « > 1 . Let these points have coordinates (w,,v,) , i = l...n in 
source digital image 700, and coordinates {x„y,), i = \...n in source digital image 
702. Consider that the exposure value recorded at the point in source digital 
image 700 is /,' , and the exposure value recorded at the point in source digital 
image 702 is 7," . Now, the aforementioned relation must hold for each point in 

1 0 the overlapping pixel region 704 . Therefore, the focal length / 71 0 can be found 
by minimizing some error measure. A typical error measure is sum of squared 
errors (SSE). Using SSE, the following function would be minimized: 




The minimum of r(j) can be found by one of a variety of different 
15 techniques; for example, nonlinear least squares techniques such as the 

Levenberg-Marquardt methods (Fletcher, "Practical Methods of Optimization", 
2"*^ Ed., John Wiley & Sons, 1987, pp. 100-1 19), or line search algorithms 
(Fletcher, pp. 33-40). 

All of the aforementioned formulas and equations can be applied 
20 when the image pixel values are proportional to the exposure values falling onto 
the image planes. If image pixel values are proportional to the logarithm of the 
exposure values (as is the case if the image pixel values are encoded in the 
nonlinear encoding of ERIMM), then all of the aforementioned formulas must be 
modified to replace cos'^(«) with 41og(cos(»)), where • indicates the argument of the 
25 cosine function. 

In some instances, the overall exposure level of each source digital 
image 700 and 702 can differ. In these cases, the light falloff and the factor 
describing the overall difference in exposure levels can be simultaneously 
determined automatically without the knowledge of the focal length; however. 
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two distinct points in the overlapping pixel region 704 are required. Copending 

USSN (EK Docket 83516/THC) filed by Cahill et al. November 5, 2001, 

details a technique for automatically determining the factor describing overall 
difference in exposure levels between multiple images, but that technique may not 
be robust if there is any significant falloff on at least one of the source images. 
Consider that the exposure value recorded at the point in the overlapping pixel 
region of source digital image 700 is 1;' , and the exposure value recorded at the 
corresponding point of source digital image 702 is Ij" , and that i>2. Then, the 
following relation must hold: 




for i = 1 ...n , where h is the factor describing the overall difference in exposure 
levels. Since l/ , Ij", u,, v;, Xi,and yj are known, the focallength f and the 
exposure factor h can be found by minimizing some error measure. A typical 
error measure is sum of squared errors (SSE). Using SSE, the following function 
would be minimized: 

r(f , h) = cos^^tan-^(^f-yiif+vf jj-Mi' oos^^tm-'i^r' ff^^ ■ 

The minimum of r(f , h) can be found by one of a variety of different nonlinear 
least squares techniques, for example, the aforementioned Levenberg-Marquardt 
methods. 

As in the case where the overall exposure characteristics of the 
source images are the same, if the image pixel values are not proportional to 
exposure values, but rather to the logarithm of the exposure values, the above 
relation becomes: 

li" -41og[^cos[^tan-l^f-^^/^^f^jjj = h+Ii' -41og|^cos[^tan-l(^f-l^uf +vf jjj , 
and the corresponding function to minimize is: 
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Kf'h) = |;Jli" -li' -h + 41og[^cos[^taii-lj^f-l^uf +vf jjj-41og|^cos[^taii-l[^f-l^xf +y? 

Referring next to Fig. 8, the process of modifying the source digital 
image 800 to compensate for light falloff is illustrated. A light falloff 

5 compensation mask 802 is generated and applied to the source digital image to 
form the adjusted source digital image 804. The compensation mask 802 can 
either be added to or multiplied by the source image 800 to fomi the adjusted 
source digital image 804. If the source image pixel values are proportional to 
exposure values, the value of the mask at pixel position (m,v) (with (o,o) being the 

1 0 center of the mask) is given by: 

mask(u, v) = j^cos'^j^tan'^j^rWu^ + v^ 

and the mask 802 is multiplied with the source digital image 800 to form the 
adjusted source digital image 804. If the source digital image pixel values are 
proportional to the logarithm of the exposure values, the value of the mask at pixel 
15 position (m,v) (with (o,o) being the center of the mask) is given by: 

mask(u, v) = -41og|^cos'*|^taa~^|^f~Wu^ 

and the mask 802 is added to the source digital image 800 to form the adjusted 
source digital image 804. 

Referring next to Fig. 9, we show a plot 900 of the pixel values in 

20 the overlap region of the second source digital 902 versus the pixel values of the 
overlap region of the first source digital image 904. If the pixel values in the 
overlap regions are identical, the resulting plot would yield the identity line 906. 
In the case that the difference between the pixel values of the two images is a 
constant, the resulting plot would yield the line 908, which differs at each value 

25 by a constant amount 910. The step 304 of modifying at least one of the source 
digital images by a linear exposure transform would then comprise applying the 
constant amount 910 to each pixel in the first source digital image. One example 
of when a linear exposure transform would be constant is when the pixel values of 
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the source digital images are in the nonlinearly encoded Extended Reference 
Input Medium Metric. The constant coefficient of the hnear exposure transform 

can be estimated by a linear least squares technique (see Lawson et al., Solving 
Least Squares Problems, SIAM, 1995, pp. 107-133) that minimizes the error 
5 between the pixel values in the overlap region of the second source digital image 
and the transformed pixel values in the overlap region of the first source digital 
image. 



estimated, but rather computed directly from the shutter speed and F-number of 
10 the lei^ aperture. If the shutter speed and F-number of the lens aperture are 
known (for example, if they are stored in meta-data associated with the source 
digital image at the time of capture), they can be used to estimate the constant 
offset between source digital images whose pixel values are related to the original 
log exposure values. If the shutter speed (in seconds) and F-number of the lens 
1 5 aperture for the first image are Ti and Fi respectively, and the shutter speed (in 
seconds) and F-number of the lens aperture for the second image are T2 and F2 
respectively, then the constant offset between the log exposure values is given by: 



and this constant offset can be added to the pixel values in the first source digital 
20 image. 

Referring next to Fig. 10, we show a plot 1000 of the pixel values 
in the overlap region of the second source digital 1002 versus the pixel values of 
the overlap region of the first source digital image 1004. If the pixel values in the 
overlap regions are identical, the resulting plot would yield the identity line 1006. 

25 In the case that the difference between the two images is a linear transformation, 
the resulting plot would yield the line 1008, which differs at each value by an 
amount 1010 that varies linearly with the pixel value of the first source digital 
image. The step 304 of modifying at least one of the source digital images by a 
linear exposure transform would then comprise applying the varying amount 1010 

30 to each pixel in the first source digital image. One example of when a linear 

exposure transform would contain a nontrivial linear term is when the pixel values 



In another embodiment, the linear exposure transforms are not 
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of the source digital images are in the Extended Reference Input Medium Metric. 
The linear and constant coefficients of the linear exposure transform can be 
estimated by a linear least squares technique as described above with reference to 
Fig. 9. 

Referring next to Fig. 1 1, the adjusted source digital images 1100 
are combined in the overlap region 1104 by a feathering scheme, weighted 
averages, or some other blending technique known in the art, to form a composite 
digital image 1106. In one embodiment, a pixel 1102 in the overlap region 1104 
is assigned a value based on a weighted average of the pixel values from both 
adjusted source digital images 1100; the weights are based on its composite 
digital image 1106 to the edges of the adjusted source digital images 1100. 

Referring next to Fig. 12, according to a further embodiment of the 
present invention, at least two source digital images are provided 1200 to the 
processing system 10. The pixel values of at least one of the source digital images 
are modified 1202 by a radial exposure transform so that any light falloflf present 
in the source digital images is compensated, yielding a set of adjusted source 
digital images. The adjusted source digital images are then combined 1204 by a 
feathering scheme, weighted averages, or some other blending technique known in 
the art, to form a composite digital image 1206. The pixel values of the composite 
digital image are then converted into an output device compatible color space 
1208. The output device compatible color space can be chosen for any of a 
variety of output scenarios; for example, video display, photographic print, ink-jet 
print, or any other output device. 

Referring finally to Figs. 13A and 13B, at least one of the source 
digital image files 1300 may contain meta-data 1304 in addition to the image data 
1302. Such meta-data 1304 could include the metric transform 1306, the shutter 
speed 1308 at which the image was captured, the f-number 1310 of the aperture 
when the image was captured, the focal length 1312 when the image was 
captured, a flash indicator 1314 to indicate the use of the flash when the image 
was captured, or any other information pertinent to the pedigree of the source 
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digital image. The meta-data can be used to directly compute the linear 
transformations as described above. 

The invention has been described in detail with particular reference 
to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the spirit and scope of the invention. 
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PARTS LIST 



10 


digital image processing system 


12 


digital image processing computer 


14 


network 


15 


image capture device 


16 


digital image store 


18 


high resolution color monitor 


20 


hard copy output printer 


21 


keyboard and trackball 


200 


provide source digital images step 


202 


modify source digital images step 


204 


combine adjusted source digital images step 


206 


composite digital image 


300 


provide source digital images step 


302 


modify source digital images with radial exposure transform step 


304 


modify source digital images with linear exposure transform step 


306 


combine adjusted source digital images step 


308 


composite digital image 


400 


source digital images 


402 


overlap regions 


500 


source digital image 


502 


metric transform 


504 


apply metric transform step 


506 


transformed source digital image 


508 


matrix transform 


510 


gamma compensation lookup table 


600 


focal length / 


602 


point (u,v) 


604 


angle 6 


606 


image center 


608 


source digital image 


700 


source digital image 


702 


source digital image 


704 


overlapping pixel region 


706 


image center 
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708 


image center 


710 


focal length / 


714 


point 


800 


source digital image 


802 


light falloff compensation mask 


804 


adjusted source digital image 


900 


plot of relationship between pixel values of overlap region 


902 


second image values 


904 


first image values 


906 


identity line 


908 


actual line 


910 


constant offset 


1000 


plot of relationship between pixel values of overlap region 


1002 


second image values 


1004 


first image values 


1006 


identity line 


1008 


actual line 


1010 


linear offset 


1100 


adjusted source digital images 


1102 


pixel 


1104 


overlap region 


1106 


composite digital image 


1200 


provide source digital images step 


1202 


modify source digital images step 


1204 


combine adjusted source digital images step 


1206 


composite digital image 


1208 


transform pixel values step 


1300 


source digital image file 


1302 


image data 


1304 


meta-data 


1306 


metric transform 


1308 


shutter speed 


1310 


f-number 


1312 


focal length 


1314 


flash indicator 



